Personalizing medical conditions with augmented reality

ABSTRACT

Augmented reality is used to simulate the impact of medical conditions on body parts and other objects within images taken of the objects. The simulations enable a user to see how a medical condition can affect the user by dynamically simulating the impact of the medical condition on captured images of body parts associated with the user in real-time. A user can select different medical conditions that are associated with different body parts. These objects are then identified within images containing the body parts using image recognition algorithms and/or user input. Thereafter, the images are modified so as to render the body parts as though the body parts were being impacted by the medical condition. The modifications are made by blending image data of the captured image with condition image data available to the processing system.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/747,595 filed on Jan. 23, 2013, entitled “PERSONALIZING MEDICALCONDITIONS WITH AUGMENTED REALITY,” which issued as U.S. Pat. No.9,256,962 on Feb. 9, 2016, and which application is expresslyincorporated herein by reference in its entirety.

BACKGROUND

1. The Field of the Invention

The present invention is generally related to computerized systems andmethods for simulating medical conditions and, even more particularly,for personalizing the visual display of medical conditions withaugmented reality.

2. The Relevant Technology

Existing computer interfaces enable users to access a variety ofinformation, including information about medical conditions. Forexample, computer interfaces enable a user to submit a query for medicalinformation and to thereafter view the results of their query displayedas text and images. Some systems are also capable of renderinginformation in rich formats, such as animations and interactivemodeling.

A new type of interfacing, comprising augmented reality, enables animage of an environment to be captured from a camera and to be augmentedor modified with simulated or supplemental imagery that makes it appearas though the environment actually contains the simulated orsupplemental imagery. In some instances, the processes associated withaugmented reality can be performed in real-time.

Despite these advances, however, there are currently no systemsavailable for enabling a user to see how a particular medical conditionwill affect the user, in real-time, with augmented reality.

BRIEF SUMMARY OF THE INVENTION

The present invention extends to methods, systems, and computer programproducts for dynamically simulating medical conditions on capturedimages of anatomical objects in real-time to thereby personalize themedical conditions with augmented reality.

In some embodiments, a user interacts with interfaces of a computingsystem to identify a medical condition associated with at least one bodypart or other anatomical object. The system is also used to access anddisplay an image of the corresponding anatomical object.

The anatomical object is also detected and recognized by the system withone or more image recognition algorithms such that the system canprovide an indication that the anatomical object has been identified byhighlighting the anatomical object in the displayed image and/or bymodifying the interfacing that is rendered to the user.

In response to subsequent user input, the identified medical conditionis rendered with or simulated on the image of the anatomical object topersonalize the medical condition with augmented reality. In oneembodiment, this is done by modifying the displayed image of theanatomical object with supplemental imagery associated with theidentified medical condition. The displayed image of the anatomicalobject can be modified by overlaying, blending, replacing, or otherwisechanging the captured imagery of the anatomical object with thesupplemental imagery associated with the identified medical condition.

In some embodiments, the steps associated with rendering the medicalcondition are performed iteratively and/or as an animation in real-time,while imagery of the anatomical object continues to be accessed anddisplayed by the processing system(s).

Using the interfaces and systems of the invention, as described herein,it is possible to personalize the visual presentation of medicalconditions and to further enhance learning associated with the medicalconditions.

These and other objects and features of the present invention willbecome more fully apparent from the following description and appendedclaims, or may be learned by the practice of the invention as set forthhereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

To further clarify the above and other advantages and features of thepresent invention, a more particular description of the invention willbe rendered by reference to specific embodiments thereof which areillustrated in the appended drawings. It is appreciated that thesedrawings depict only illustrated embodiments of the invention and aretherefore not to be considered limiting of its scope. The invention willbe described and explained with additional specificity and detailthrough the use of the accompanying drawings in which:

FIG. 1 illustrates a user holding a computing device that includes adisplay screen that is presently displaying an image of a portion of theuser's face;

FIG. 2 illustrates an embodiment similar to FIG. 1, wherein one of theuser's eyes is fully displayed on the display screen and wherein thateye has been highlighted;

FIG. 3 illustrates an embodiment similar to FIG. 2, wherein the displayof the highlighted eye has been modified to reflect a medical conditionassociated with the eye;

FIG. 4 illustrates a back perspective view of a user holding a computingdevice with a camera and a display screen displaying an image of theuser being captured by the camera, wherein the user's mouth has beenhighlighted;

FIG. 5 illustrates another view of the computing device of FIG. 4,wherein the image of the user's mouth is zoomed in on within thedisplayed image on the display screen;

FIG. 6 illustrates an additional view of the computing device, as shownin FIG. 4, wherein the user's mouth is displayed with augmented realitymodifications in such a way as to reflect a medical condition associatedwith the mouth and wherein a play bar reflects a relative progression orseverity associated with the medical condition;

FIG. 7 illustrates another view of a computing device, as shown in FIGS.5-6, wherein the display of the user's mouth is further modified toreflect a different severity or stage of the medical condition andwherein the play bar is replaced by an interface control for selectablycontrolling the simulated display of different stages of the medicalcondition via augmented reality;

FIG. 8 illustrates a flow diagram of various acts that are associatedwith the methods of the invention; and

FIG. 9 illustrates one example of a computing environment that can beutilized to implement aspects of the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

User interfaces are provided and utilized by the invention topersonalize the manner in which medical conditions are rendered byelectronic devices.

According to some embodiments, a user is able to see how a medicalcondition can affect the user by dynamically simulating the impact ofthe medical condition on captured images of body parts associated withthe user in real-time.

Initially, a user interacts with a computing system to identify amedical condition or other context associated with an anatomical object(e.g., a body part). The system is then used to identify and detect theanatomical object within a captured image. The system provides anindication that the anatomical object has been detected by highlightingthe anatomical object within the displayed image and/or by modifying theinterfacing that the system renders to the user, such as, for example,by displaying a user interface control on the system display screen thatis operable to initiate a rendering of the medical condition on theanatomical object.

In response to subsequent user input, such as the user selecting theinterface control, the medical condition is simulated with the displayof the anatomical object in the form of augmented reality. In oneembodiment, this is done by overlaying, blending, replacing, orotherwise modifying the captured image of the anatomical object withsupplemental imagery associated with the identified medical condition.

In some embodiments, the rendering of the medical condition is performediteratively and/or as an animation, in real-time, while imagery of theanatomical object continues to be captured and displayed by the systemas a stream of images.

Using the interfaces and systems of the invention, as described herein,it is possible to visually render the impact of a medical condition on auser in a personalized and intuitive manner.

It is noted that the term ‘medical condition,’ as used herein, caninclude negative medical conditions (e.g., degenerative conditionsassociated with unhealthy tissues and the presence of diseases or otherharmful agents/objects), as well as positive medical conditions (e.g.,treatments or other conditions that are associated with beneficialagents/objects). The medical conditions can also correspond to discretestages of treatment, degeneration, passage of time, personalized medicalrecords, diagnosis, and/or other medical information.

The term ‘medical condition’ is sometimes referred to herein as a‘medical context.’ While the term ‘medical condition’ or ‘medicalcontext’ can apply, in singular form, to only a single medicalcondition, it will be appreciated that the term ‘medical condition’ or‘medical context’ can also apply to and include a plurality of differentmedical conditions that are related, but which are capable of beingcategorized independently.

The medical conditions or contexts described herein are associated withdifferent anatomical structures by correlation tables and other datastructures. These anatomical structures can include any body parts(e.g., eyes, nose, mouth, hair, teeth, gums, tongue, fingers,fingernails, hands, toes, feet, organs, bones, or any other body parts)or any combinations of body parts.

The terms ‘computing system,’ ‘system,’ ‘computing device,’ and‘device,’ which are used interchangeably herein, all refer to computingsystems that include physical computing hardware, such as a physicalcomputing processor, a camera, a display screen, a storage device (e.g.,computer memory and/or disk storage), and/or other computer hardware asdescribed in more detail with reference to FIG. 9. In many embodiments,the computing systems are mobile devices (e.g., phones, tablets,notebook computers, portable gaming device, etc.) that are utilized tointeract with the user interfaces of the invention. In otherembodiments, the computing systems comprise stationary devices (e.g.,desktop computers, gaming consoles, televisions, servers, kiosks, etc.)or a combination of stationary and mobile devices.

Preferably, although not necessarily, the computing systems havemulti-touch screens that are operable to receive and process touch inputand to display relevant output, although other input and output devicescan also be used.

Personalizing Medical Conditions with Augmented Reality

FIG. 1 illustrates an a user holding a computing device 100 that isdisplaying an image 110 on a display screen 120. The image 110 is beingcaptured in real-time, in response to the device 100 operating in animage capture mode (e.g., camera mode). The camera capturing the image110 is a back-facing camera that is not presently visible. The backfacing camera is similar to the front-facing camera 130, which ispresently illustrated as a lens. Other camera hardware that is operableto capture the image 110 is housed within the device 100.

In some embodiments, the device 100 is put into a camera modeautomatically in response to a user selecting a medical context (e.g., amedical condition) from a menu interface (not presently shown). It willbe appreciated that different types of user interface menus can beutilized to enable a user to select a medical context or condition.Various non-limiting examples of such interface menus are shown and/ordescribed in the following documents, which documents are allincorporated herein by reference in their entireties: U.S. patentapplication Ser. No. 13/093,272, filed Apr. 25, 2011; U.S. patentapplication Ser. No. 13/167,610, filed Jun. 23, 2011; U.S. patentapplication Ser. No. 13/167,600, filed Jun. 23, 2011; U.S. patentapplication Ser. No. 13/237,530, filed Sept. 20, 2011; U.S. patentapplication Ser. No. 13/477,794, filed May 22, 2012; U.S. patentapplication Ser. No. 13/663,820, filed Oct. 30, 2012; and U.S. patentapplication Ser. No. 13/720,196, filed Dec. 19, 2012.

In some embodiments, the device is manually put into a camera mode(before or after the medical context is identified) in response to auser selecting a camera mode button (not presently shown), which can bea physical button on the device or a virtual OS (operating system)interface control button associated with the device.

While in the camera mode, the systems of the invention utilize imagerecognition algorithms to detect one or more anatomical objectsdisplayed within the images received by the device that are associatedwith the identified medical context. Image recognition algorithms thatrely on spatial relationships, light spectrum analysis, light contrastanalysis, and other distinguishing features or identifiable markersdetected in an image are well-known to those of skill in the art. Knownimage recognition algorithms are currently used, for example, toidentify faces and eyes.

The present invention utilizes data structures, such as tables andindexes that correlate anatomical objects with the image recognitiondata (e.g., spatial dimensions, color and light analysis and/or otherdistinguishing image features) as well as metadata that can beassociated with recorded images. Then, when an image is captured orotherwise received and analyzed with the image recognition algorithmsfor the presence of the distinguishing image features and/or metadata,it is possible to use the correlation tables and other data structuresto identify the anatomical objects that are associated with the detectedimage recognition data in the stored tables and data structures.

While the systems of the invention can be configured to identify anycombination and quantity of anatomical features, it is preferable thatthe device will specifically search for and detect/identify theanatomical objects that are specifically associated with a particularmedical context, based on relationships specified in the stored tablesand data structures.

By way of example, in one embodiment, the medical conditionconjunctivitis is associated with optical structures in the datastructures, such that when the user identifies the medical conditionconjunctivitis, the system will specifically search for, and in someinstances only search for, anatomical objects associated withconjunctivitis, such as an eye and possibly tissue surrounding the eye(e.g., eyelids).

By way of another example, the data structures can link the condition ofgingival recession (receding gums) with an anatomical object comprisingthe mouth (in the broadest context), and/or one or more oral components,such as teeth, gums, dental roots, and/or other oral tissues. Each ofthese related anatomical objects are further associated with thecorresponding image recognition data in the same data structure or adifferent data structure. In this manner, when a user selects acondition associated with receding gums, the inventive systems willspecifically search for and identify the anatomical objects (mouth,teeth, gums, and so forth) that have been associated with gingivitis inthe stored correlation tables. The system will also search the same ordifferent data structure(s) for any corresponding image recognition datathat can be used by the image recognition algorithms to identify thepresence of the corresponding anatomical object(s) in the image datacaptured by or otherwise received by the systems of the invention.

With regard to the foregoing, it is noted that it is possible to havedifferent medical conditions associated with the same anatomical objectsin the data structures. Similarly, a single medical condition/contextcan actually be associated with a plurality of discrete medicalconditions. For instance, a condition or context of dental disease canbe associated with receding gums, gingivitis, periodontitis, as well asother conditions that are each associated with combinations of the sameanatomical objects by the correlation tables and data structures.

While many of the foregoing descriptions specifically relate to theidentification of anatomical objects exclusively with the use of imagerecognition algorithms, it will be appreciated that the invention alsoextends to embodiments in which user input, such as touch input detectedat the display screen while an anatomical object is displayed, toindependently select or cooperatively (with the image recognitionalgorithms) facilitate the identification of the anatomical object.

It is also noted that the identification or detection of the anatomicalobjects within an image can be performed exclusively or partially by thedevice and/or another system that is in communication with the deviceand that receives and processes the image data from the device.

Attention is now directed to FIG. 2, which illustrates an embodiment inwhich a user has already identified a medical context, such asconjunctivitis, from one or more user interface menus (not shown) and inwhich the device is operating in a camera mode automatically in responseto the user selecting the medical context or in response to the userselecting an augmented reality button (not presently shown) that isdisplayed on the device in response to the user selecting the medicalcondition from the user interface menus.

As shown, the device 200 is displaying an image 210 that is beingcaptured by a back facing camera (not shown). The image 210 is displayedon a display screen 220 along with highlighting, comprising a frame 240,that is displayed around the image of the user's eye.

According to some embodiments, the highlighting of the object (e.g., theeye) is performed automatically by the device 200 in camera mode, inresponse to the device 200 analyzing the captured image data for thedistinguishing features that have been associated with the identifiedmedical context by the correlation tables and, thereafter, upondetecting the presence of the anatomical object having thedistinguishing features within the captured image that is now displayed.

The highlighting of an anatomical object that has beendetected/identified/selected can be performed by framing the object, asshown. In other embodiments, not presently shown, the highlighting ofthe object is performed by changing a color of the object (e.g., byshading or other color changes), by changing a size of the object (e.g.,by zooming in or enlarging the object), by outlining one or morecomponents of the object, by annotating the object with supplementalmarkers or text, and/or by any other highlighting technique.

According to some embodiments, the device highlights the anatomicalobject substantially as soon as it becomes completely visible within thecaptured image data that is displayed on the display screen, as shown inFIG. 2. This is contrasted with the illustration of FIG. 1, wherein theeye is only partially displayed and is not, therefore, highlighted. Itother embodiments, the anatomical object is highlighted as soon as itbecomes identified, even if it is only partially visible within thedisplay screen. For instance, the portion of the eye that is illustratedin FIG. 1 can be highlighted in this embodiment as soon as it becomesidentified using the image recognition algorithms described above.

In other embodiments, the device refrains from highlighting theidentified anatomical object, even when the anatomical object isassociated with the identified medical condition.

A user can tilt, rotate, move, or zoom their device and/or device camerauntil a suitable image is captured that includes an anatomical objectthat is determined to be associated with the identified medical context.In some embodiments, the user can also switch between different camerasof the device (e.g., front facing, back facing and other cameras) inorder to best capture the desired imagery containing the anatomicalobject(s) associated with the identified medical condition.

In some embodiments, the detection of the anatomical object associatedwith a medical context further triggers the display of an augmentedreality control, such as the virtual play control 250 shown in FIG. 2,with or without the aforementioned highlighting.

The augmented reality control is operative, when selected, to initiatethe rendering of the medical condition as a simulation that is used tomodify the displayed imagery of the detected anatomical object viaaugmented reality processing.

FIG. 3, for instance, reflects an embodiment similar to FIGS. 1 and 2,wherein the illustrated device 300 is used to render an image 310 on adisplay screen 320, wherein a medical context comprising conjunctivitishas been identified in response to user input, and wherein acorresponding anatomical object (e.g., an eye) has been detected andhighlighted by a display frame 340 within the displayed image 310.

As further shown, an augmented reality control 350 is also displayed,which is operable to initiate the augmented reality simulation of themedical condition on the displayed image 310. As illustrated, thiscontrol 350 already been selected by touch input rendered at the displayscreen 320. This input has initiated the rendering of the medicalcondition (conjunctivitis) on the user's eye tissues as augmentedreality within the displayed image 310. For instance, as shown, the eyeis currently displayed with mucopurulent discharge 360 and an inflamedor otherwise agitated conjunctiva 362. This simulation of the medicalcondition is rendered on the displayed image 310 by modifying portionsof the captured image that are associated with the identified anatomicalobject (e.g., the eye).

The rendering of the condition, via augmented reality processing, can beperformed with any known image rendering techniques, including blending,masking, overlaying, replacing, coloring, or otherwise modifyingportions of the captured image data, such as the portions of the imagedata that are associated with the anatomical object.

The captured image data is preferably modified with condition image datathat is stored or remotely accessed by the system and that is correlatedwith the anatomical objects that are identified within the capturedimages. For instance, the condition image data correlated withconjunctivitis can include the displayed imagery of the mucopurulentdischarge 360 and inflamed conjuncitiva tissue 362.

It will be appreciated that the augmented reality processing performedby the systems of the invention also includes accessing, positioning,aligning, and configuring the size, shape, and coloring of storedcondition image data in such a way that the condition image data will beproperly positioned, aligned, sized, and colored with the anatomicalobject to enhance the realism of the simulation of the medical conditionwith the anatomical object in the displayed image. Any known techniquescan be used for positioning, aligning, sizing, and coloring the storedcondition image data, including but not limited to, known techniques formapping, scaling, positioning, rotating and other processing of imagedata. The amount and type of positioning, aligning, sizing and coloringapplied to the condition image data will be directly based on theanalysis performed on the captured image data and detected anatomicalstructure(s). In some embodiments, the positioning, aligning, sizing,and coloring applied to the condition image data is also based on aselected/identified severity of the condition to be rendered with theanatomical structure.

In some embodiments, the device 300 performs the image processingassociated with the augmented reality (e.g., the augmented realityprocessing). In other embodiments, the device 300 sends the image datato another system that performs a portion or all of the augmentedreality processing and which sends modified image data and/orinstructions to the device 300 which are operable to render thecondition as augmented reality on the displayed anatomical object.

The simulating of the medical condition can consist of performing of theaugmented reality a single time, for a static image that is rendered bythe device 300. Alternatively, the simulating of the medical conditionis performed iteratively, as an animation, in real-time and on-the-flyas new image data continues to be captured/accessed and rendered by thedevice 300. When animated, the rendering of the condition, is preferablyperformed by modifying each of a plurality of display frames that aredisplayed by the device to reflect the impact of the medical conditionon the corresponding anatomical object in the display, and such that theanatomical object continues to be displayed with the condition even whenmovement of the device causes the anatomical object to be displayedwithin a different portion of the display screen 320. This isillustrated, for example, as the position/rotation of the device changesfrom FIGS. 2-3, the highlighting frame (240, 340) moves with themovement of the eye within the displayed image. Similarly, themucopurulent discharge 360 and inflamed conjuncitiva tissue 362 alsomove with the eye within the displayed image 310 as the device isrotated, tilted or otherwise moved. This embodiment is also beneficialfor enhancing the realism of the simulated condition when the usertilts, rotates or otherwise moves their body (and the correspondinganatomical object), such that the rendering of the simulated conditionis modified in size, shape, color, and other configurations to match themovements and changes to the user's body elements where the condition issimulated within the displayed imagery.

According to some embodiments, the augmented reality simulation presentsonly a single stage or severity of the medical condition by makingsubstantially the same or similar condition modifications to the imageof the anatomical object within each display frame of the image data. Inother embodiments, the augmented reality includes presenting thecondition as a dynamic condition which changes over time. For instance,in one embodiment, the conjunctivitis shown in FIG. 3 represents arelatively severe state of conjunctivitis, which is shown only afterfirst showing the eye affected with one or more less severe states ofconjunctivitis, wherein the displayed image smoothly transitions betweenthe different stages of the condition over time. In this regard, themedical condition is displayed as a life-cycle animation by applyingdifferent modifications to the same portions of the displayed anatomicalobject within different sequential display frames of the displayed imagedata over time.

While the modification of the image data can occur in substantiallyreal-time, as the image data is captured and rendered on-the-fly, itshould also be appreciated that the modifications of the image data canalso occur, in some embodiments, on a delayed basis. For instance,recorded video and other image data can be modified at a substantiallylater time than it was recorded (e.g., a minute later, several minuteslater, or hours, days, weeks months or years later). Likewise, themodified image data can be rendered at the same time the modificationsto the image data occurs, or at a substantially later time than when themodification to the image data occurs (e.g., a minute later, severalminutes later, or hours, days, weeks months or years later).

Terminating the simulation of the medical condition can occur in directin response to user input. For instance, once the augmented reality isinitiated, the augmented reality control 350 can be changed into a stopbutton or a pause button (not presently shown), which is operable (whenselected) to stop the augmented reality processing. A separate stop orpause button can also be displayed and used to perform similarfunctionality.

Terminating the simulation of the medical condition can also occur inresponse to a timer control (e.g., predetermined passage of time) and/orupon the completion of a computer process (e.g, an animationlife-cycle).

Alternatively, moving the device so that the anatomical object is nolonger displayed within the display screen 320 can also trigger thetermination of the augmented reality processing.

FIG. 4 illustrates a back perspective view of a user holding a computingdevice 400 with a camera 430 and a display screen 420 that is displayingan image 410 of the user. The image 410 is being captured by the camera430 in response to the device 400 being placed in an image capture modefollowing the user's selection of a condition from a user interface menu(not shown) and/or in response to the user selecting an augmentedreality control link from a user interface (not shown).

In the present embodiment, the user has selected a gum disease condition(e.g., receding gums). Accordingly, the device has highlighted theuser's mouth with a displayed highlighting frame 440. In otherembodiments, only the gums are highlighted with outlining or anotherform of highlighting. In yet other embodiments, the mouth and gums arenot highlighted at all, even when identified.

Once the anatomical objects associated with the selected condition aredetected within the image the device 400 displays an augmented realitycontrol 450 for initiating the simulation of the selected condition. Thedisplay of the augmented reality control 450 and the highlighting frame440 (when present) can also operate as an indicator that the anatomicalobject(s) associated with the selected condition have been identifiedand that the device is ready to render the augmented reality simulationof the medical condition. Once the augmented reality control 450 isselected, the simulation of the selected medical condition is initiated,as described above.

FIGS. 5-6 illustrate different periods of time after the augmentedreality control 450 has been selected and different corresponding statesof the augmented reality simulation corresponding to different stages orseventies of the medical condition. In these embodiments, the image isalso zoomed in around the detected anatomical object (e.g., the mouth).This zooming can be performed automatically by the device, in responseto the user input selecting the augmented reality control 450 or inresponse to detecting the anatomical object. In other embodiments, thedevice does not zoom in on the image. Instead, the enlarged imageresults from the user repositioning the device (e.g., moving the devicecamera closer to the user's face and mouth).

As illustrated, the device 500 of FIG. 5 includes a display 520 that isdisplaying an image 510 of the user's mouth with a highlighting frame540 around the mouth. The image 510 is also displayed with a play bar550 that includes an indicator 552 that reflects a relative progressionof the selected medical condition (e.g., receding gums). In someembodiments, the indicator 552 is a passive indicator. In otherembodiments, the indicator 552 is an active control that is operable,when selected and moved by user input, to adjust the relativeprogression or severity of the condition that is being displayed.

FIG. 6 illustrates a device 600 that is similar to the device 500 ofFIG. 5. For instance, the device 600 includes an image 610 of the user'smouth with a highlighting frame 640 around the user's mouth whichresulted from a user selecting the medical condition associated with themouth (e.g., gum disease) and in response to the device detecting thepresence of the mouth within the image that is being displayed on thedisplay screen 620. The image 610 is also displayed with a play bar 650that has an indicator 652 reflecting the relative progression of ananimation associated with the medical condition.

In the current embodiment, the augmented reality applied to the image610 includes simulating receding gums by modifying the presentation ofthe displayed gums 660 on the image of the user's mouth. As reflected,the indicator 652 on the play bar 650 is about halfway between the startand end of the play bar 650, indicating the severity of the condition isabout a medium level of severity and/or that the simulation of themedical condition is about halfway through its animation cycle.

For the current state of the condition, the simulation includesdisplaying the user's gums 660 in the live camera display as though thegums 660 have receded to such a degree that they have exposed the roots662 of some of the teeth and such that the gums have become somewhatinflamed (which can include a change in size and color). This simulationoccurs by replacing or modifying portions of the gums from the capturedimage data with new condition image data that is appropriatelypositioned, aligned, sized, shaped, and colored to match thecorresponding condition and stage of the condition, as described above.The new condition image data used to modify the image is changed orreplaced with different new condition image data as the state of thecondition changes and/or as the device captures different images inwhich the user's mouth is displayed in a different position.

The referenced animation cycle can progress automatically from start tofinish, once started. Alternatively, the cycle can be controlled inresponse to a user manually manipulating the relative placement of theindicator 652 on the play bar 650.

With regard to the foregoing embodiment, it will be appreciated that theanimation cycle can comprise a partial cycle, starting with a firststate or severity of the condition and ending with a different state orseverity, as suggested with regard to FIG. 6. Alternatively, theanimation cycle can also comprise a complete cycle that progresses froma first state to one or more subsequent states and then back to thefirst state by rendering the one or more subsequent states in reverseorder.

FIG. 7 illustrates another embodiment for controlling the simulation ofthe selected medical condition. In FIG. 7, a device 700 is used todisplay an image 710 of a mouth on a display screen 720, as generallydescribed above. Likewise, the mouth is highlighted by a frame 740surrounding the mouth. However, instead of providing a play bar, thepresent embodiment includes an augmented reality control 750 that has aplurality of selectable controls (752, 754, 756) which, when selected,advance the condition to a corresponding state of severity. Forinstance, selection of control 752 will result in the user's mouth beingbe displayed as rendered in FIG. 5, selection of control 754 will resultin the user's mouth being displayed as rendered in FIG. 6, and whereasselection of control 756 (as selected) results in the mouth beingdisplayed as rendered in FIG. 7, with severely exposed roots 762 andseverely inflamed gums 760.

Attention will now be directed to FIG. 8 which illustrates a flowdiagram 800 of various acts that are associated with methods of theinvention, which correspond to the functionality described above.

As shown, the methods of the invention can include the act of or stepfor identifying a medical context 810. This aspect of the invention canalso include a plurality of related acts, such as receiving user inputfor identifying or selecting a medical condition 812 and receiving inputfor identifying and selecting an anatomical object associated with themedical condition 814.

The act of receiving input for identifying and selecting the medicalcondition 812 can occur when a user selects an interface menu object(with touch input or other input) corresponding to the medicalcondition. In other embodiments, the user can provide textualinformation or other typed input that relates to the medical condition.In some embodiments, the medical condition can also be identified andselected in response to input received from a third party, such as adoctor or a system that is capable of performing a diagnosis of acondition. In any embodiments, the input is preferably received at thedevice that is utilized to display the images of the anatomical objectswith the simulations of the identified medical conditions.

As noted above, the medical conditions that are selected or identifiedare associated with one or more corresponding anatomical objects. Theseanatomical objects are also associated with distinguishing visualproperties or characteristics that can be identified in images thatinclude the anatomical objects.

The methods of the invention also include the initiation of a cameramode or an image capture mode 820. This can occur, in some embodiments,automatically in response to the device identifying the medicalcondition/context and the associated anatomical objects that relate tothe identified medical condition.

In camera mode or image capture mode, the device captures or otherwiseaccesses an image 830 containing the associated anatomical object(s).Preferably, the images containing the anatomical objects are captured bythe same device that displays the images and at a time that iscontemporaneous with the time that the images are being captured, suchas in a live video display format. In other words, the image is capturedby a camera attached to the device and the image that is capturedincludes an anatomical object that is visible to the camera at the timethe image is captured. However, the images can also be captured byanother device and provided to the display device. While the images arepreferably displayed in real-time, as they are captured, they can alsobe displayed at a much later time. Accordingly, the act of initiatingthe image capture mode 820 of the device, to capture the images can alsobe viewed more broadly as initiating the accessing of an image by thedevice. Similarly, the acts of capturing and displaying the capturedimage(s) can be broadly viewed, in some embodiments, as accessing anddisplaying an image containing the anatomical object(s) associated withthe identified medical condition.

The methods of the invention also include an act of detecting thepresence of anatomical object(s) within the captured/accessed image(s)840. This act can be performed with the use of user input thatidentifies the anatomical object(s) and/or in response to the analysisperformed with the image recognition algorithms of the captured imageand with the use of the data structures that correlate anatomicalobjects to image characteristics, as described above in reference toFIGS. 1-7.

Once an anatomical object is detected within the captured image data, anindication of that detection can be made (850) by highlighting theanatomical object and/or by modifying the interfacing displayed with theimage containing the anatomical object, as described above in referenceto FIGS. 1-7.

In some embodiments, the detection of the anatomical object(s) 840 andthe indicating of the detection of the anatomical object(s) 850 includesdistinguishing the identified anatomical object from one or more otheranatomical objects that are also present in the captured image(s) whichcan be associated with the same medical condition or a different medicalcondition. For example, the captured image may include the presence oftwo eyes and a mouth. However, if the condition is associated with themouth, the systems of the invention will refrain from highlighting ordetecting the presence of the eyes by only searching for featuresmatching the visual characteristics of the mouth. Alternatively, thesystem may identify, but simply refrain from highlighting the eyes. Inanother example, the systems may also detect and/or highlight only oneof the two eyes when the condition is associated with the eyes.

The medical condition associated with the anatomical object is alsorendered with the captured/access image (870) in response to user inputthat initiates the rendering/simulation (860), as described above withrespect to FIGS. 1-7. In some embodiments, the input initiating therendering comprises a user selecting a condition severity and/or themanipulation of a play button or an augmented reality control, asdescribed above. This input can also be received after the simulation ofthe condition with the anatomical object has already begun. Forinstance, the user can provide input that adjusts the severity of acondition after the condition is already being rendered with theanatomical object.

Preferably, although not necessarily, the simulation of the medicalcondition is rendered as occurring on a surface element (e.g., skin, orsurface tissue) of the anatomical object that is visible to the nakedeye and the camera, while that surface element is simultaneously visibleto the camera and at least partially displayed on the display screen. Itis also preferable, although not necessary, for the simulation of themedical condition to occur at substantially the same time that the imageis captured and processed for the augmented reality, such as, forexample, within less than a few seconds of the image capture and evenmore preferably in less than a single second of the image capture.However, the simulation can also occur many seconds, minutes, hours,days and even weeks after the image capture, as described above.

While the acts described above can be performed in the illustratedsequence, it will be appreciated that these acts can also be performedin a different sequence as well. For instance, by way of example, theidentification of the medical context/condition (810, 812) can occursubsequent to the acts of displaying the captured image (830) and eventhe detecting the presence of the anatomical objects (840). In fact, thedetected presence of the anatomical objects (840), in some embodiments,can automatically trigger the identification/selection of the medicalcondition (812) which is ultimately determined to be correlated to theidentified medical condition.

Accordingly, while various specific examples have been provided herein,it will be appreciated that the invention extends to any combination ofthe functionality, interfaces, systems and methods described herein.Furthermore, while many of the embodiments have been described withregard to degenerative or negative medical conditions, it will also beappreciated that the identified/simulated conditions can also correspondto treatments that simulate healthy tissues in the place of capturedimage data that reflects unhealthy tissues, for example.

It will also be appreciated that the anatomical objects described abovecan include anatomical objects comprising human body parts as well asbody parts of animals and plant parts. The referenced anatomical objectscan also include published reproductions of human, animal or plantparts, such as images displayed in a magazine or other publication,which are capable of being detected and displayed by the systems of theinvention.

Computing Environment(s)

Embodiments of the present invention may comprise or utilize specialpurpose or general-purpose computing devices that include computerhardware, such as, for example, one or more processors and systemmemory, as discussed in greater detail below. Embodiments within thescope of the present invention also include physical and othercomputer-readable and recordable type media for storingcomputer-executable instructions and/or data structures. Suchcomputer-readable recordable media can be any available media that canbe accessed by a general purpose or special purpose computer system.Computer-readable media that store computer-executable instructionsaccording to the invention are recordable-type storage media or otherphysical computer storage media (devices) that are distinguished frommerely transitory carrier waves.

Computer-readable media that carry computer-executable instructions aretransmission media. Thus, by way of example, and not limitation,embodiments of the invention can comprise at least two distinctlydifferent kinds of computer-readable media: computer storage media(devices) and transmission media.

Computer storage media (devices) includes RAM, ROM, EEPROM, CD-ROM,DVD-ROM, HD-DVD, BLU-RAY or other optical disk storage, magnetic diskstorage or other magnetic storage devices, or any other medium which canbe used to store desired program code means in the form ofcomputer-executable instructions or data structures and which can beaccessed by a general purpose or special purpose computer and which arerecorded on one or more recordable type medium (device).

A “network” is defined as one or more data links or communicationchannels that enable the transport of electronic data between computersystems and/or modules and/or other electronic devices. When informationis transferred or provided over a network or another communicationsconnection or channel (either hardwired, wireless, or a combination ofhardwired or wireless) to a computer, the computer properly views theconnection as a transmission medium. Transmissions media can include anetwork and/or data links which can be used to carry or desired programcode means in the form of computer-executable instructions or datastructures and which can be accessed by a general purpose or specialpurpose computer. Combinations of the above should also be includedwithin the scope of computer-readable media.

Further, upon reaching various computer system components, program codemeans in the form of computer-executable instructions or data structurescan be transferred automatically from transmission media to computerstorage media (devices) or vice versa. For example, computer-executableinstructions or data structures received over a network or data link canbe buffered in RAM within a network interface module (e.g., a “NIC”),and then eventually transferred to computer system RAM and/or to lessvolatile computer storage media (devices) at a computer system. Thus, itshould be understood that computer storage media (devices) can beincluded in computer system components that also (or even primarily)utilize transmission media.

Computer-executable instructions comprise, for example, instructions anddata which, when executed at one or more processor, cause one or moregeneral purpose computer, special purpose computer, or special purposeprocessing device to perform a certain function or group of functions.The computer executable instructions may be, for example, binaries,intermediate format instructions such as assembly language, or evensource code. Although the subject matter has been described in languagespecific to structural features and/or methodological acts, it is to beunderstood that the subject matter defined in the appended claims is notnecessarily limited to the described features or acts described herein.Rather, the described features and acts are disclosed as example formsof implementing the claims.

Those skilled in the art will appreciate that the invention may bepracticed in network computing environments with many types of computersystem configurations, including personal computers, desktop computers,laptop/notebook computers, message processors, hand-held devices,multi-processor systems, microprocessor-based or programmable consumerelectronics, network PCs, minicomputers, mainframe computers, tablets,mobile telephones, PDAs, pagers, routers, switches, and the like. Theinvention may also be practiced in distributed and cloud systemenvironments where local and remote computer systems, which are linked(either by hardwired data links, wireless data links, or by acombination of hardwired and wireless data links) through a network,both perform tasks. In a distributed system environment, program modulesmay be located in both local and remote memory storage devices.

FIG. 9 illustrates an exemplary computing environment 900 that can beused to present the user interfaces of the invention, to facilitate userinteraction with anatomical structures rendered on the user interfaces,and to demonstrably reflect the impact of various conditions andtreatments on those anatomical structures.

As shown, the computing environment 900 includes one or more computingsystems 910 in communication with one or more other devices and/orsystems 920 through one or more network connections 930. The othersystems and devices can be output and input devices, servers, proxies,or any other systems or devices. The network connections 930 can bewired and/or wireless connections and can include any combination ofLocal Area Network (“LAN”) connections, Wide Area Network (“WAN”)connections, including the Internet and one or more proxy servers.

As illustrated, the computing system 910 is configured with a storage950 that has stored computer-executable instructions for implementingthe methods of the invention. The storage 950 also stores medical recorddata (e.g., annotations, medical diagnosis data, condition correlationtables and data structures, image data associated with variousanatomical objects and conditions, image rendering instructions forperforming the augmented reality processing and image blending, and soforth), as well as image recognition algorithms that are operable toidentify/detect anatomical objects from the image data that is capturedby the device camera 992, based on the stored medical record data and/orother known metrics and distinguishing visual characteristics associatedwith the anatomical objects, as described above with reference to FIGS.1-7.

The storage 950 also store modules, such as the user interface (UI)module 960 and the communication module 970, which each comprisecomputer-executable instructions and data structures for implementingaspects of the invention.

The communication module 970, for instance, includes computer-executableinstructions that, when executed by one or more processors 980 areoperable to facilitate wireless and/or wired communications through thenetwork connections 930 to access or transmit data associated with theinterfaces described herein, including the interfaces themselves. Thecommunication modules are also configured to encrypt and decrypt dataand to perform authentication of user and system credentials.

The interface module 960 includes computer-executable instructions that,when executed by the one or more processors 980 are operable to generateand/or present the user interfaces described above. Interface module 960also provides computer-executable instructions that, when executed, areoperable to detect and process user input and to perform thefunctionality described above, such as, but not limited to modifying thedisplay of the anatomical objects by replacing/augmenting and/orblending the display data with supplemental image data associated withdifferent medical conditions.

The data stored at storage 950 can also include any of the interfacesand data described in reference to the following patent applications,each of which is incorporated herein by reference its entirety, and suchthat the present invention can incorporate any combination offunctionality described in this document as well as any of thefunctionality described in the following documents: U.S. patentapplication Ser. No. 13/093,272, filed Apr. 25, 2011; U.S. patentapplication Ser. No. 13/167,610, filed Jun. 23, 2011; U.S. patentapplication Ser. No. 13/167,600, filed Jun. 23, 2011; U.S. patentapplication Ser. No. 13/237,530, filed Sept. 20, 2011; U.S. patentapplication Ser. No. 13/477,794, filed May 22, 2012; U.S. patentapplication Ser. No. 13/663,820, filed Oct. 30, 2012; and U.S. patentapplication Ser. No. 13/720,196, filed Dec. 19, 2012.

The storage 950 can comprise, but is not limited to, non-volatile diskstorage and volatile memory. It will also be appreciated that thestorage 950 can be distributed among a plurality of different devices orsystems, including the other illustrated system(s)/device(s) 920, anddoes not necessarily need to be constrained to a single physical device.In some embodiments, however, the storage 950 is limited to localstorage of the computing system 910.

In some embodiments, the computing system 910 comprises a wireless cellphone, a tablet computer, a notebook computer, a PDA, and/or any othertype of smart device having a display screen 994, camera 992, speakersand other hardware 990 for rendering image data, audio data, and/ortextual data to a user via the interfaces (960, 970) of the system(910), for example. In some embodiments, the hardware 990 of the system910 includes a display screen 994 that is a multi-touch screen capableof receiving touch input.

The computing systems of the invention can include mobile devices (e.g.,phones, tablets, notebook computers, portable gaming device, etc.) andstationary device (e.g., desktop computers, gaming consoles,televisions, servers, kiosks, etc.).

It will be appreciated that the system hardware 990 can include anyoutput device (e.g., display screen, speakers, printer, etc.), as wellas any input device (keyboard, mouse, microphone, touchpad, etc.).

Each of the other connected systems/devices 920 also includes hardware,processor and storage components similar to those described above withrespect to system 910.

The present invention may be embodied in other specific forms withoutdeparting from its spirit or essential characteristics. The describedembodiments are to be considered in all respects only as illustrativeand not restrictive. The scope of the invention is, therefore, indicatedby the appended claims rather than by the foregoing description. Allchanges which come within the meaning and range of equivalency of theclaims are to be embraced within their scope.

What is claimed is:
 1. A storage device having storedcomputer-executable instructions which, when executed by at least onecomputing processor, implement a method for dynamically modifying animage containing an anatomical object that is displayed on a displayscreen of a computing system that includes a camera that is used tocapture the image that is displayed and modified according to themethod, the method comprising: using the camera of the computing systemto capture the image containing the anatomical object; using thecomputing system to automatically detect the presence of the anatomicalobject within the captured image, the anatomical object being associatedwith a medical condition; displaying the captured image on a displayscreen associated with the computing system, the captured image beingrendered in real-time with the automatically detected anatomical object;and an act of the computer system, in response to detecting user inputmanipulating a control for rendering a simulation at the computingsystem, rendering the simulation of the medical condition associatedwith the automatically detected anatomical object at the display screen,wherein the simulation dynamically reflects a simulated impact of themedical condition associated with the corresponding anatomical object bydisplaying a progressive stage of the medical condition from the captureimage.
 2. The storage device recited in claim 1, wherein the computingsystem is a mobile computing system, wherein the mobile computing systemincludes the camera used to capture the image containing the anatomicalobject, and wherein the mobile computing system also includes thedisplay screen wherein the simulation is rendered.
 3. The storage devicerecited in claim 1, wherein the method further includes receiving userinput manipulating the control at the computing system subsequent todisplaying the captured image and prior to rendering the simulation. 4.The storage device recited in claim 3, wherein the simulation isrendered in response to receiving user input selecting a play button. 5.The storage device recited in claim 1, wherein simulation of the medicalcondition consists of modifying a single display frame rendering thecaptured image as a static image.
 6. The storage device recited in claim1, wherein the simulation of the medical condition comprises themodifying of a plurality of sequential display frames used to render aplurality of the captured images on the display screen, including thecaptured image, as an animation.
 7. The storage device recited in claim6, wherein similar modifications are made to correspondingly relatedportions of the plurality of sequential display frames, thecorrespondingly related portions of the plurality of sequential displayframes rendering similar image data associated with the anatomicalobject at different portions on the display screen while the anatomicalobject is moved relative to the camera.
 8. The storage device recited inclaim 6, wherein different modifications are made to correspondinglysimilar portions of the plurality of sequential display frames toreflect a dynamic change in the simulation of the medical condition,while the anatomical object remains in a same position relative to thecamera such that the correspondingly similar portions of the pluralityof sequential display frames rendering similar image data associatedwith the anatomical object.
 9. The storage device recited in claim 1,wherein the medical condition is a degenerative or harmful condition,and wherein the simulation dynamically reflects the simulated impact ofthe medical condition over time by displaying different, progressivestages of the medical condition over time and transitioning between thedifferent, progressive stages of the medical condition over time to showthe progression of the medical condition over time.
 10. The storagedevice recited in claim 9, wherein the medical condition is at least oneof conjunctivitis, gingival recession, gingivitis, or periodontitis. 11.The storage device recited in claim 1, wherein the medical condition isa treatment or beneficial condition, and wherein the simulationdynamically reflects the simulated impact of the medical condition overtime by displaying different, progressive stages of the medicalcondition over time and transitioning between the different, progressivestages of the medical condition over time to show the progression of themedical condition over time.
 12. The storage device recited in claim 1,wherein the rendering of the simulation of the medical condition withthe anatomical object includes blending image data of the captured imagewith stored condition image data associated with the identifiedcondition.
 13. The storage device recited in claim 1, wherein theindicator comprises a highlighting element that highlights theautomatically detected anatomical object in the captured image.
 14. Thestorage device recited in claim 1, wherein the method further includesreceiving user input for identifying the medical condition.
 15. Thestorage device recited in claim 1, wherein the simulated impact of themedical condition on the corresponding automatically detected anatomicalobject is based on a user selected severity of the medical condition.16. The storage device recited in claim 1, wherein the medical conditionis at least one of conjunctivitis, gingival recession, gingivitis, orperiodontitis.
 17. The storage device recited in claim 1, wherein thestorage device further stores one or more data structures correlatingthe identified medical condition with the anatomical object and theanatomical object with image characteristics associated with theanatomical object.
 18. The storage device recited in claim 1, whereinthe anatomical object comprises at least one of an eye and a mouth. 19.A computer implemented method for dynamically modifying an image of ananatomical object that is displayed on a display screen of a computingsystem, the method comprising: obtaining an image containing ananatomical object; using the computing system to automatically detectthe presence of the anatomical object within the captured image, theanatomical object being associated with a medical condition; displayingthe captured image on a display screen associated with the computingsystem, the captured image being rendered with the automaticallydetected anatomical object and an indicator that the automaticallydetected anatomical object has been automatically detected; and an actof the computer system, in response to detecting user input manipulatinga control for rendering a simulation at the computing system, renderingthe simulation of the medical condition associated with theautomatically detected anatomical object at the display screen, whereinthe simulation dynamically reflects a simulated impact of the medicalcondition over time by displaying, with the simulation, differentprogressive stages of the medical condition over time and transitioningbetween the different progressive stages of the medical condition overtime to show the progression of the medical condition over time.
 20. Acomputing device comprising: at least one hardware processor; a displayscreen; and one or more storage media having stored computer-executableinstructions which, when executed by the at least one hardwareprocessor, implement a method for dynamically modifying an image of abody part that is displayed on a display screen associated with thecomputing device, the method comprising: obtaining an image containingan anatomical object comprising a body part; detecting the presence ofthe body part within the image, the anatomical object being associatedwith a medical condition; displaying the image on a display screenassociated with the computing system, the image being rendered with thebody part in a way that highlights the body part and in such a way as toindicate that the body part has been detected within the image; andrendering augmented reality on the display screen, the augmented realitycomprising a dynamic simulation of the medical condition that modifiesat least a portion of the captured image of the body part and that isrendered as an animation with a plurality of different display frameswhich reflects the simulated impact of the medical condition, over time,on the image of the body part.